Guideline: Tips - Estimate The Effort (AST)

Sometimes a total budget is imposed by the client for testing. This has to be spread across the test phases and the characteristic/object part combinations. Some of the techniques described in Estimation Techniques (Ratios and Work Breakdown Structure, inparticular) provide assistance here. It should also be examined whether the budget is adequate.

The following tips are useful, but much depends on the experience of the test manager:
1. It is best to create an estimate by summing up the characteristic/depth-of-testing estimates (e.g. in PAT, a light performance test + a thorough security test = 120 + 200 hours = 320 hours).
2. Another option is to evaluate the total budget using a rule-of-thumb summing up for test levels: 15% of the total project budget of 5,000 hours for the ST, 20% for the AT = 750 and 1,000 hours respectively.
3. A third option is to employ a standard allocation formula for characteristics, established on the basis of a number of experiences. An example is to give Functionality, at risk category B, 70% of the total budget, at A, 80% and at C, 60%. You can also do this with a fixed number of hours, e.g. User-friendliness could be given 70 hours at category C, and up to 130 at category A (for an average system). This will make it easier to assess the real value of the individual estimates per test level /characteristic.
4. Compare the allocated budget with the budgets and total hours spent in the course of comparable exercises in the past, both in the organisation itself and if possible in other, comparable, organisations.
  
  The estimate should be assessed for real value and should come out at around the level of the total assigned budget. Otherwise, adjustments will be necessary, by opting for a higher total budget, or testing fewer characteristics and/or testing in less depth.
  
  The figures used above are realistic examples. More information can be found in Estimation Techniques.
A depth of testing level of ●●● for a characteristic in a particular test level (e.g. Functionality in the ST), means that the system should be tested in more depth than average, not that the entire system should be tested in the greatest possible depth. See also the comment under Test Strategy.
The creation of an estimate for the test has a wide margin of uncertainty. It is important that the test manager make it clear to the stakeholders that the estimate is based on a number of assumptions and may therefore have to be revised later. A possible solution is the use of uncertainty margins. At the beginning of the test, the margin would be, for example, around 40%; at the start of the test execution this becomes around 25%, and somewhere in the middle it becomes around 10%.
The link between estimate and depth of testing (using specific test techniques) is opaque. How much extra time does, for example, the application of the elementary comparison test require as against the data combination test? Few past figures are available for this, and much is done on the basis of the test manager’s experience and intuition.
Various other factors (the quality of the testers, of the test object and test basis, test environment and test tools) can also exert signifi cant influence on the estimate. These factors are either not known at the time the estimate is made or their effect on the estimate is very unclear. The test manager has to make assumptions here, and if necessary include them as assumptions in the plan, and most certainly should evaluate the assumptions as soon as possible.
It is difficult to ‘sell’ the required maintenance test effort to management. A general ‘testing image’ problem is that testing costs too much in management’s view. With testing during maintenance, that is reinforced by the fact that testing has a relatively big share in the maintenance effort – up to as much as 80%. This is partly because the total test costs consist of fixed and variable costs. Fixed costs refer to, for example, the effort required to prepare the test environment, or the execution of a ‘standard’ regression test; variable costs refer to, for example, the preparation and testing of implemented changes. With the testing of a small change, the percentage of fixed costs is high, since, irrespective of the size of the change, the environment must always be prepared and the regression test run. The greater the changes become, the more the fixed costs decrease in relative terms. For example, with the testing of a change, a 4-hour regression test is always run. If the testing of a change takes 8 hours in total, the fixed testing time amounts to 50% (4/8). If the testing of one or more changes takes a total of 40 hours, then this decreases to 10% (4/40). In general, the share of testing (fixed+variable) lies between 35% - 80% of all the maintenance activities. It is up to the test manager to make this clear to the client and to put the case for the importance (to the testing) of bigger, controlled releases, in which many changes are bundled, over the implementation of a constant procession of small changes.